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GitHub - oduwsdl/memgato 
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MemGator 
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A Memento Aggregator CLI and Server in Go. 

Features 

• The binary (available for various platforms) can be used as the CLI or run as a Web Service 

• Results available in three formats - Link/JSON/CDXJ 

• TimeMap, TimeGate, and Memento (redirect or description) endpoints 

• Optional streaming of benchmarks over Server-Sent Events (SSE) for realtime visualization and monitoring 

• Good API parity with the main Memento Aggregator service 

• Concurrent - Splits every session in subtasks for parallel execution 

• Parallel - Utilizes all the available CPUs 

• Custom archive list (a local JSON file or a remote URL) - a sample JSON is included in the repository 

• Probability based archive prioritization and limit 

• Three levels of customizable timeouts for greater control over remote requests 

• Customizable logging and profiling in CDXJ format 

• Customizable endpoint URLs - helpful in load-balancing 

• Customizable User-Agent to be sent to each archive and User-Agent spoofing 

• Configurable archive failure detection and automatic hibernation 

• CORS support to make it easy to use it from JavaScript clients 

• Memento count exposed in the header that can be retrieved via head request 

• Docker friendly - An image available as ibnesayeed/memgator 

• Sensible defaults - Batteries included, but replaceable 

Sawood Alam (@ibnesayeed) https://qithub.com/oduwsdl/memqator 
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MementoDamage 


Home API Faq Help 


How well is your webpage archived? 



http://odu.edu/compsci 
Damage = 0.024 



http://clarkjolley.com 

Damage = 0.44014 



Audemus Jura Noslra Defenders 

I ' IVf Dote Dt/end Our Rights") 


http://alguard.state.al.us 
Damage = 0.96800 


Check the damage of your page 


http://www.somesite.com 


Check » 


Note : The memento damage calculation will work on live webpages, but was designed to 
evaluate archived webpages or mementos. Discover mementos using Time Travel 


Erika Siregar (@erikaris) http://memento-damaqe.cs.odu.edu/ 





I#: Mat Kelly, PhD Student of Co. x 
G © www.cs.odu.edu/~mkelly/ 
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95 mementos available. 


Select a Memento 


View Archive Page To... 


List mementos by: Dropdown Drilldown 

W — I' 

Old Dominion University 

Current/Upcoming Projects 

• Research and paper development on various aspects of personal web archiving. 

• Continued development of WARCreate . Web Archiving Integration Layer (WAIL) , and Mink for NEH Digital Humanities 
Implementation Grant . 

• Continued development of Interplanetary Wavback (ipwb) . 



Upcoming/Current Projects 
Papers, Posters & Presentations 
Recognition & Participation 
teaching 


Papers, Posters & Presentations All Only Peer-Reviewed Only Papers Only Journals 

• Mat Kelly, Sawood Alam, Michael L. Nelson, and Michele C. Weigle. "Interplanetary Wayback: Peer-To-Peer Permanence of Web — r ece n /§ oftwa r e Pr o ect s 
Archives," In Proceedings of the International Conference on Theory and Practice of Digital Libraries (TPDL). Hannover, J 


Germany, September 2016, pp. 411-416. ( PDF . BibTeX) 

Mat Kelly, Sawood Alam, Michael L. Nelson, and Michele C. Weigle, "Interplanetary Wayback: The Permanent Web Archive," At 
the Web Archiving and Digital Libraries Workshop (WADL 2016). Newark, NJ, June 2016. 

Sawood Alam, Mat Kelly, and Michael L. Nelson, "Interplanetary Wayback: The Permanent Web Archive," In Proceedings of the 
IEEE/ ACM Joint Conference on Digital Libraries (JCDL). Newark, NJ, June 2016, pp. 273-274. (PDF . BibTeXl 
Mat Kelly, "Exploring Aggregation of Personal, Private, and Institutional Web Archives," Presented At Archives Unleashed 2.0: 
Web Archive Datathon, 2016 June 15. ( PPTX 1 

Mat Kelly, "A Framework for Aggregating Private and Public Web Archives," Bulletin of IEEE Technical Committee on Digital 
Libraries (IEEE-TCDL), Vol. 1 1 , No. 3, December 2015. (PDF . BibJfeX) 

Mat Kelly, "A Framework for Aggregating Private and Public Web Archives," at the ACM/IEEE Joint Conference on Digital 
Libraries (JCDL). Doctoral Consortium. Knoxville, TN, June 2015. ( PDF . BibTeX - ) 

Wesley Jordan, Mat Kelly, Justin F. Brunelle, Laura Vobrak, Michele C. Weigle and Michael L. Nelson, "Mobile Mink: Merging 
Mobile and Desktop Archived Webs," In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL). Knoxville, 
TN, June 2015, pp. 243-244, Best Poster Award. ( PDF . BibTeX - ) 

Mat Kelly, Michael L. Nelson, and Michele C. Weigle, "Visualizing Digital Collections of Web Archives," Web Archiving 
Collaboration: New Tools and Models', 2015 June 4; New York City, NY. (PPTX) 

Justin F. Brunelle, Mat Kelly, Hany SalahEldeen, Michele C. Weigle and Michael L. Nelson, "Not All Mementos Are Created 
Equal: Measuring the Impact of Missing Resources," International Journal of Digital Libraries (IJDL), 16(3), pp. 283-301 . May 
2015. (article . BibTeX - ) 

Mat Kelly, "Facilitation of the A Posteriori Replication of Web Published Satellite Imagery", Virginia Space Grant Consortium 


Contact 


c.v. 


• Or 


Justin F. Brunelle, Mat Mat Kelly (@machawk1) https://qithub.com/machawk1/Mink 

International Journal < 
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Archive Now (archivenow) 

A Tool To Push Web Resources Into Web Archives 

Archive Now (archivenow) currently is configured to push resources into four public web archives. You can easily 
add more archives by writing a new archive handler (e.g., ia_handler.py) and place it inside the folder "handlers". 

As explained below, this library can be used through: 

• CLI 

• A Web Service 

• A Docker Container 

• Python 

Installing 

The latest release of archivenow can be installed using pip: 

$ pip install archivenow 


The latest development version containing changes not yet released can be installed from source: 

$ git clone git(5)github.com:maturban/archivenow.git 
$ cd archivenow 

$ pip install -r requirements.txt 
$ pip install ./ 

Mohamed Aturban (@maturban1 ) https://qithub.com/oduwsdl/archivenow 
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WARCreate 

Google Chrome Extension 

"Create WARC files from any webpage" 

What is it? 

WARCreate is a Google Chrome extension that allows a user to create a Web ARChive 
(WARC) file from any browseable webpage. The resulting files can then be used with 
other tools like the Internet Archive 's open source Wavback Machine . The tool is an 
evolving product with the end result pushing toward being a personal web archiving 
solution for those that wish to securely archive their metadata in a standardize way. 

Where Can I Download It? 

WARCreate can be downloaded from the Chrome Web Store . 

What Can I Do with Generated WARCs? 

Software is needed to replay the WARCs by nature of the format. 

I recommend using Web Archiving Integration Laver (WAIL), a software suite that 
came about because of WARCreate. 

I Found A Bug! Where Do I Report It? 

Send all bug reports to Mat Kelly 


Mat Kelly (@machawk1 ) http://warcreate.com/ 
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Web Archiving Integration Layer (WAIL) 


"One-Click User Instigated Preservation" 


Web Archiving Integration Layer (WAIL) 

"One-Click User Instigated Preservation" 

Web Archiving Integration Layer (WAIL) is a graphical user interface (GUI) atop multiple web archiving tools 
intended to be used as an easy way for anyone to preserve and replay web pages. Tools included and accessible 
through the GUI are Heritrix 3.2.0 and PyWb 0.33.0. 

More information about the motivations behind WAIL see the Motivations section in the projects wiki. 

This work is supported by the National Endowment for the Humanities (NEH), through Digital Humanities grants 
HD-51670-13 and HK-50181-14 
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Wail Electron 
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John Berlin ((a)johnaberlin) https://aithub.com/N0taN3rd/wail 
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chrome web store 


rhodewarriors@gmail.com ▼ $ 




Local Memory Project 

offered by localmemoryproiect 
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( 0 ) 


Search Tools 


7 users 


OVERVIEW 


REVIEWS 


RELATED 


LOCAL MEMORY PROJECT 


SaBAggcsrogtoaafrlimrmegajKureij 


Stories for query: “protesters and police", captured: Tue, Sep 13, 2016. 12:53 PM EDT, for 
23529 (Norfolk VA, USA). Archive: search 


1 . Hampton Roads Messenger, Newspaper, (2.98 miles, VA - USA) Exclude (saved/archived) 


2. Inside Business, Newspaper, (2.98 miles, VA - USA) Exclude (saved/archived) 


p VIRGINIAN-PILOT KIMBERLY PIEF 


P VIRGINIAN-PILOT MIKE HIXENBAI 




P VIRGINIAN-PILOT ROBYN SIDERSK1 

[If 


P VIRGINIAN-PILOT JONATHAN EOWAR 


Portsmouth council meeting 
punctuated by hymns, praise, 
tension ... (Jul 12. 2016) 



A moment ot healing in Portsmouth. 
A black protester hugging a while 
(Jul 13. 2016) 


Gathering outside the Scope arena in 
Norfolk protests police shootings (Jul 
9. 2016) 



P VIRGINIAN-PILOT AMIR VERA 



Local Black Lives Matter leaders, police 

praise one another after ... (Jul 11. 

2016) 


Protesters of police shootings organize 
in Portsmouth, take to the ... (Jul 11. 
2016) 


On Sal, rosy marring, a smal group gatherso na 


oo»oe ovaf and ms protesters lor . . 


negotiated Strain «6 


P VIRGINIAN-PILOT ASSOCIATED PRE 




Protesters remained peaceful. 


Protesters of police shootings hug 



P VIRGINIAN-PILOT JANE COWAN 

si 
BV 
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Protest against police-involved 



+ ADD TO CHROME 
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Compatible with your device 


Build collection of local events from 
local sources 


* nj-itnr- Affnr 


A Website 
O Report Abuse 


Additional Information 

Version: 0.0.0.5 
Updated: February 17, 2017 
Size: 1.38MiB 
I an puapp: Fn^lteh 


Alexander Nwala (@acnwala) http://www.localmemorv.org/ (joint with 

Harvard} 
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The Local Memory Project provides tools to 
build collections (#localmemory) of stories 
for local events from local news sources. 

This extension helps users 
build/archive/share collections about local 
events or stories collected from local news 
media outlets. For example, given a zip code 
"701 1 5” and a query "flooding," the 
extension will collect local stories from 
media outlets that sen/e New Orleans, LA, by 
searching Google. The extension also lets 
you search Google news and filter based on 
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Interplanetary Wayback (ipwb) 

Peer-To-Peer Permanence of Web Archives 


1 build 

passing 


■ 



[ pypi 

vO. 2017. 2. 18.2104 


Interplanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the 
contents of WARC files into the IPFS network. IPFS is a peer-to-peer content-addressable file system that 
inherently allows deduplication and facilitates opt-in replication, ipwb splits the header and payload of WARC 
response records before disseminating into IPFS to leverage the deduplication, builds a CDXJ index with references 
to the IPFS hashes returns, and combines the header and payload from IPFS at the time of replay. 

Interplanetary Wayback primarily consists of two scripts: 

• ipwb/indexer.py - archival indexing script that takes the path to a WARC input, extracts the FITTP headers, 

FITTP payload (response body), and relevant parts of the WARC-response record header from the WARC 

Sawood Alam (@ibnesayeed) & Mat Kelly (@machawk1) https://qithub.com/oduwsdl/ipwb 
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Carbon Dating The Web 

Predict the Birthday of a Webpage! 


www.cs.odu.edu 


Carbon Date! 



"self" : "http: //cd. cs . odu.edu/cd?url=www.cs . odu.edu" , 

"URI" : "http : //www.cs . odu.edu" , 

"Estimated Creation Date": " 1997-03-24T17 : 29 : 34 " , 

"Backlinks " : " 1997-07-02T13 : 41 : 36 " , 

"Last Modified": 

"Bitly.com" : " 2011-08-31T14 : 05 : 22 " , 

"Twitter.com" : "2017-02-19T16:21:10", 

"Bing.com": 

"Google.com" : "2005-05-28T00 : 00 : 00 " , 

"Pubdate tag": 

"Archives": [ 

[ 

"Earliest" , 

" 1997-03-2 4T 17:29:34" 

], 

[ 

"By_Archive" , 

{ 

"http: //archive. is/ 199706 06 105 03 9 /http: //www.cs .odu . edu/" 
"http: / /arquivo .pt/wayback/20091223043049/http: //www.cs.oi 
"http: / /webcitation.org/query?id=1327284 086752784 " : "2012- 
"http: //web . archive .org/web/ 19 97 10 1020 1632/http ://www.cs.i 
"http : / /web . archive .bibalex . org: 80 /web/ 2 00104 14022512/htt] 

> 

] 

] 

} 


Zetan Li http://cd.cs.odu.edu/ 
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@ What Did It Look Like? 
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What Did It Look Like? 


We randomly choose some web sites and see what they looked like through the years. 

To nominate a URL for inclusion, tweet "#whatdiditlooklike URL". Follow @ wdill for daily updates. 


2007 



Ihtts //nnn egr vc« edu/ Oo 

OCT 

JAN 

FEB 

Cloi' X I 

MMUMi 

-1 

324 capture* 1 

i mi atti’a^ h m IhiJihfe «l J i i 

◄ 

2005 

4 

2007 

► 

2008 

Hep'J 


biomedical 

chemical A life science 
computer science 
electrical A computer 
mechanical 
dean's office 


• Current students • Future students • Academic programs ■ Faculty & staff • Research/interdisciplinary initiatives 


Alexander Nwala (@acnwala) http://whatdiditlooklike.mennentoweb.orq/ 




I Can Haz Memento (@icanh x 


© «" 




G i Twitter, Inc. [US] https://twitter.com/icanhazmemento 


☆ © 


^ri 



I Can Haz Memento 

©icanhazmemento 

Use #icanhazmemento to request links to 
URLs archived near the time they were 
shared in your tweet. Via ©WebSciDL. 

dH] Joined July 2015 


Tweet to I Can Haz Memento 


£ 8 Followers you know 

££JIHA 


Tweets Tweets & replies 

•4> In reply to Michael L. Nelson 
'V ~1p I Can Haz Memento Jicanhazmemento • Jan 25 

.@phonedude_mln, Your newly archived page: 
timetravel.mementoweb.org/memento/201701 ... (Internet Archive ...). 
Other versions: timetravel.mementoweb.org/list/201 701 250. . . 

4 > t* ¥ 

In reply to Michael L. Nelson 

'V -jP I Can Haz Memento Mcanhazmemento ■ Jan 24 

,@phonedude_mln, Your newly archived page: 
timetravel.mementoweb.org/memento/201701 ... (VM Brasseuron 
Tw...). Other versions: timetravel.mementoweb.org/list/201 701250... 

4% In reply to Michael L. Nelson 

Ml I Can Haz Memento icanhazmemento ■ Jan 24 


wt Alexander Nwala (@acnwala) https://twitter.com/icanhazmemento 
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To detect the off-topic for a collection 

• Using collection id from Archive-lt 

python detect_off_topic.py -i [collection_id] 

For example: 


GitHub - yasmina85/OffTop x 




i GitHub, Inc. [US] https://github.com/yasmina85/OffTopic-Detection 
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python detect_off_topic . py -i 1860 


• Using collection uri from Archive-lt 


python detect_off_topic . py -r [collection_uri] 

For example: 

python detect_of f_topic . py -r https://www.archive-it.org/collections/1860 

• To check off-topic for one timemap 

python detect_off_topic.py -t [timemap_uri] 

For example: 


python detect_off_topic . py -t https://wayback.archive-it.Org/2358/timemap/link/http://hamdeensabahy.com/ 


• The default will list the off-topic mementos on the screen, if you want to forward the result to another file 

python detect_of f_topic . py -i [collection_id] -o [filename] 


• To change the threshold value 

owthnn dofnrt «">££ 5 L ml 1 rn- +• -i on i A 1 + h ^ 

Yasmin AINoamany (@yasmina_anwar) https://qithub.com/vasmina85/OffTopic-Detection 


Shortest Possible PhD Topic Summaries 

(+ 1 link for more info) 



Temporal Violations, Archive Profiling, 

Cold Spots 


Scott Ainsworth (@Galsondor) 

- Detecting temporal violations in archival replay 

- http://ws-dl.bloqspot.com/2015/12/2015-12-08-evaluatinq-temporal.html 

Sawood Alam (@ibnesayeed) 

- Profiling web archives 

- http://dx.doi.ora/1 0. 1 007/s00799-01 6-01 84-4 

Lulwah Alkwai (@LulwahMA) 

- Eliminating “cold spots” in web archives 

- http://www.cs.odu.edu/~mln/pubs/icdl-2015/icdl-2015-arabic-sites.pdf 


Tampering, Storytelling, 

Private Web Archives 

Mohamed Aturban (@maturban1) 

- Detecting archival tampering 

- (will have demo at CNI Spring 2017) 

Shawn M. Jones (@shawnmjones) 

- likely to continue Yasmin AINoamany’s storytelling 
work 

- https://qithub.com/vasmina85 

Mat Kelly (@machawk1 ) 

- Integrating private and public web archives 

- http://ws-dl.bloqspot.com/2012/08/2012-08-20-nns-thesis-extensible.htnnl 


Automating Archival Collections, Page 
Transformation, Finding Alumni 


Alexander Nwala (@acnwala) 

- Bootstrapping collections via Twitter, Storify, Reddit, et al. 

- http://ws-dl.bloaspot.com/201 6/07/201 6-07-1 8-tweet-visibilitv-dvnamics-in.html 

John Berlin (@johnaberlin) 

- “It became necessary to destroy the page to save it” 

- (currently MS, will enter PhD) 

- http://ws-dl.bloqspot.com/2017/01/2017-01-2Q-cnncom-has-been-unarchivabl 

e.html 

Corren McCoy (@CorrenMcCoy) 

- Finding university alumni in social media 

- (only one not working explicitly in web archiving!) 

- http://ws-dl.bloaspot.eom/2015/1 1/2015-1 1-24-twitter-follower-analvsis-of.html 


#IA20 



#IA20 - East Coast 



http://ws-dl.bloqspot.eom/2016/1 1/2016-1 1-21-ws-dl-celebration-of-ia20.html 

https://storifv.com/michaelnelson/ws-dl-celebration-of-ia20 




